I have created a test team site with a document library name "My doc lib" and a test user who does not have access to the team site, as shown below
Here are the steps to configure the Anonymous access to the Lists and Libraries(ex., My doc Lib) and before proceding to the steps, lets login with the new user and try to access the document library
Ok, test Passed :) , Document Library Access is denied for the test user and here are the steps.Step 1: Go to the library settings of the document library
Step 2: Click Permission for this document library
Step 3: Now the permission Tools page will be displayed, but this page(Ribbon) will not display the anonymous access option, because this libary is inheriting the permission from its parent, and in this scenario it inherits permission from the Team site.
Step 4 -Stop Inheriting Permissions from the parent
Done!, now the anonymous user is able to access the document library, see top right corner.