Directory Propblem

[savantcreative]

I submitted to a free directory PR2 a few months ago on a site that had been up since December and had about 1000 backlinks but I was already on Google Page 1 position 2-3 for my keywords. Then all of a suddenly I started getting hundreds of links from this directory because they featured my site as a new listing and I appear on every page. At first I though it was great until I started seeing my serp rankings dropping slowly. Do you think this sudden influx of low quality links can be doing the damage? What would you do. I don't but links and am all white hat.
Thanks