The idea of "toxic" links seems to have become all-pervasive and is causing more harm to SEO than any perceived benefits, IMO. See this recent discussion, for instance:

Why does site with more toxic links rank better? [webmasterworld.com]

Directories have been poor links for years - they are an old means to browse content that has been completely supplanted by search as the number of available sites has grown. Does this make directories "toxic"? Of course not. It just makes them a waste of time. You might find some niche directories of relevance, but I would be shocked if you could find more than a handful in any industry.

You mention that a particular directory has been penalised. What are the symptoms of that? Has it lost it's own rankings? Been removed from results? Or has it been labelled as bad by a third party tool of some sort?

why some directories are deemed ok and some are not

If a directory is "free for all" and has no or very limited editorial control then it's essentially a very low quality user-generated site that consists mostly of links. Google has never liked such sites - the "directory" label is irrelevant.

If your site has 50 backlinks, and 49 of them come from small general directories, this wouldn't look right.

You mention that a particular directory has been penalised. What are the symptoms of that? Has it lost it's own rankings? Been removed from results? Or has it been labelled as bad by a third party tool of some sort?

It doesn't even rank top for its own brand anymore, so this set alarm bells ringing. Despite it having massive authority inbound links.

3zero -- Does it charge a "review fee" for getting a site listed in the directory? Because if so, this can be seen as a form of link selling, which is a violation of google's guidelines.

The worst toxic links are the links you never gain or blindly removed because you are too fearful. If you honestly feel a link can generate legitimate web traffic that will convert on your website it will usually be a boost to your efforts.

If you are trying to find the easiest links without any regard for their value to users then you are more likely going to have a bad day. Most directories cashed in on the link buying frenzy several years ago and produced little to no value to their users and sent little to no legitimate traffic to the link buyers. These directories are not a good idea.

There are some good niche directories that can be great to seek out (for example a professional trade association directory that is strictly controlled). They provide a unique & useful service for consumers and actually send legitimate traffic.

As goodroi points out, specific and niche directories can have some value. These days, however, directories are not generally seen as a great source... that day has come and gone.