Inconsistent download speeds with modem or ISP

I'm seeing non consistent download speeds coming out of the modem with computer hard coded to modem during normal work week days and hours EST.  Seems better download speeds during off hours or on weekends.  Wondering if the firmware update would help with this issue.  Optimum claims that there is over 500 mbps coming into the modem and this modem should be able to handle 680 mbps.  Since my computer is hard coded should I receive consistent speed thoughout the day unless the ISP has issues with bandwidth.  Why does over 500 mbps coming in get whittled down to under 40 mbps at times during the day?