Pseudo-nitzschia blooms often occur in coastal and open ocean environments, sometimes leading to the production of the neurotoxin domoic acid that can cause severe negative impacts to higher trophic levels. Increasing evidence suggests a close relationship between phytoplankton bloom and bacterial assemblages, however, the microbial composition and succession during a bloom process is unknown. Here, we investigate the bacterial assemblages before, during and after toxic and non-toxic Pseudo-nitzschia blooms to determine the patterns of bacterial succession in a natural bloom setting. Opportunistic sampling of bacterial community profiles were determined weekly at Santa Cruz Municipal Wharf by 454 pyrosequencing and analyzed together with domoic acid levels, phytoplankton community and biomass, nutrients and temperature. We asked if the bacterial communities are similar between bloom and non-bloom events and if domoic acid or the presence of toxic algal species acts as a driving force that can significantly structure phytoplankton-associated bacterial communities. We found that bacterial diversity generally increases when Pseudo-nitzschia numbers decline. Furthermore, bacterial diversity is higher when the low-DA producing P. fraudulenta dominates the algal bloom while bacterial diversity is lower when high-DA producing P. australis dominates the algal bloom, suggesting that the presence of algal toxin can structure bacterial community. We also found bloom-related succession patterns among associated bacterial groups; Gamma-proteobacteria, were dominant during low toxic P. fraudulenta blooms comprising mostly of Vibrio spp., which increased in relative abundance (6–65%) as the bloom progresses. On the other hand, Firmicutes bacteria comprising mostly of Planococcus spp. (12–86%) dominate during high toxic P. australis blooms, with the bacterial assemblage showing the same bloom-related successional patterns in three independent bloom events. Other environmental variables such as nitrate and phosphate and temperature appear to influence some low abundant bacterial groups as well. Our results suggest that phytoplankton-associated bacterial communities are strongly affected not just by phytoplankton bloom in general, but also by the type of algal species that dominates in the natural bloom.