On IRC clients the viewer list is populated from /NAMES and JOIN/PART messages. TMI does JOIN/PART messages in a batch every 30 seconds or so.
To get membership state event (NAMES, JOIN, PART, or MODE) functionality. You need to execute:
CAP REQ :twitch.tv/membership
upon connect.
/NAMES is only executed on your join to the channel, but will not list everyone if the channel is large(1000+) in which case it will only populate the moderators and streamer.
You could try polling the unofficial/unsupported TMI API, https://tmi.twitch.tv/group/user/:CHANNEL:/chatters to get the information you need.
Also, you may want to look at: Pircbot, problems with retrieving a full userList. Java - #2 by 3ventic