As I recall Spinoza is generally held to have a pretty good record when it comes to race. There is as far as I know only one passage of one letter, written to Pieter Balling in July, 1664, in which he seems to use the term 'black' in a negative way. I haven't seen the original Dutch (I've seen notes that suggest it may in fact have have been written in Spanish first, and then translated), which would be helpful, but I will cite Curley's English rendition here:
One morning, as the sky was already growing light, I woke from a very deep dream to find that the images which had come to me in my dream remained before my eyes as vividly as if the things had been true --especially the image of a certain black, scabby Brazilian whom I had never seen before. For the most part this image disappeared when, to divert myself with something else, I fixed my eyes on a book or some other object. But as soon as I turned my eyes back away from such an object without fixing my eyes attentively on anything, the same image of the same Black man appeared to me with the same vividness, alternately, until it gradually disappeared from my visual field (IV 76, 25-IV 77, 6).
Rather than handing out laurels to, or withholding them from, past thinkers on the basis of the connotations we assume must timelessly attach to words, it would be more useful to try to trace back the connotations that led the thinkers themselves to employ these words. And here there can be no doubt that Spinoza is using a stock example from the history of humoral medicine.
To be 'black' in this context is to have a surfeit of black bile or melancholy, and thus to be earthy, and thus to be associated with corpses and other supposedly dark things. In extreme cases, a melancholic person, in addition to being dark or 'muddy' him or herself, can also hallucinate corpses or, alternatively, 'black' men.
In The Anatomy of Melancholy Robert Burton attributes to Jovianus Pontanus the view that those born under the influence of Saturn are "very austere, sullen, churlish black of colour" (Mem. I, subs. 3, 397). Hercules de Saxonia is cited as holding that "these that are naturally melancholy" are "of a leaden colour or black" (ibid., 399). Burton himself adds that a certain subspecies of melancholics "are fatter than others..., of a muddy complexion" (ibid, 400).
It is precisely such people, according to Guianerius, who "think themselves dead many times, or that they see, talk with black men, dead men, spirits and goblins frequently" (ibid., 400). Gordonius writes that extreme melancholics see and talk "with black men, and converse familiarly with devils" (ibid., 402). Gentilus Fulgosus is cited as relating the story of a melancholy friend, who "'had a black man in the likeness of a soldier' who followed him wherever he was" (ibid.).
In a footnote to the citation from Guianerius, Burton distinguishes two different sorts of reactions to sudden, unexpected visions: there are those who blandiuntur, and those who territant, i.e., those who are 'terrified', which is to say, etymologically, those who are rendered earthy, of which a sign is the sudden darkening of the face. The extreme consequences of this, for people of a certain humoral disposition, is to come to believe that they themselves are corpses (a condition which has apparently been clinically confirmed in recent years).
I suspect that this particular sort of hallucination originally had no racial connotation whatever, but was thought to be the straightforward consequence of the excessive production of black bile. That blackness is associated with earth in humoral medicine (an association also widely attested in sub-Saharan ethnography too; see, e.g., Goody 1977) enables in turn the association of the melancholic condition with corpses, and the interchangeability of the figure of the black man with that of the corpse.
It is likely that between the time of Burton's work and Spinoza's letter, there is a superimposition of a racial or quasi-racial connotation upon the figure of the black man, and that the term is gradually deprived of its original humoral significance. In Spinoza's case the figure is not only racialized, but is explicitly associated with a particular geographical region; and yet, at the same time he retains the scabrousness of the old visions of corpses that interested Burton.
In any case Spinoza's hallucinated Brazilian does not appear out of nowhere; if he in fact saw it as he reports, then this was under circumstances much like those that cause me to see images from old Looney Tunes cartoons when I shut my eyes to go to sleep: a historically conditioned phantasm. And when Spinoza relates it, in the context of a letter about the nature of misleading perceptions, he is probably drawing more on a (perhaps unconscious or largely forgotten) familiarity with medical literature than he is on any negative associations that he himself attaches to blackness.