If you're lucky, you already have lots of experience with both Object Pascal (or, as it used to be called, Borland Pascal With Objects--essentially, Turbo Pascal v5.5 or later) and with Visual Basic. If you fit this description, then Delphi will be a breeze for you.
Okay, now for everyone else. In order to make full use of the Delphi environment, you have to know Pascal, you have to have some grasp of object orientation, and you have to understand event-driven programming. Once you're over those three hurdles, you've pretty much got it. See section 5 for more information.
On the other hand, most people don't need to make "full" use of the environment. If you just want to pull a simple application together that doesn't do anything too fancy, Delphi shouldn't be any harder to learn than VB--it's just that there's a whole lot more you *can* do in Delphi, which will make you feel more lost than you really are.