I don't always use it but when I do I've not had a problem with its speed. I use a 3.4GHz iMac. I just used it now in a project to test it before responding to your question and there's no observable lag/slowness. I must start using it more as its pretty awesome.
It might just be my mac and PC (both a little older) but the caching and load times on previews is excruciatingly slow. I think it just might be time to upgrade my hardware.
i use bridge ALL THE TIME, since it loads faster than finder sometimes, and since most of the files I work with are image/visual its awesome to be able to browse the thumbnails easily instead of finder previews. Sometimes the metadata info and stuff is useful for me too.
I'm pretty sure the speed is dependant on the version of bridge vs. your operating system/space.
I'm pretty sure the speed is dependant on the version of bridge vs. your operating system/space.