More than 3 years ago, I wrote a blog Load Created By, Created Date, Last Modified By, and Last Modified Date (Audit Fields) on how to upload system fields, such as: Created By, Created Date, Last Modified By, and Last Modified Date in Salesforce. Usually, we need to do this as part of the data migration activity, where those fields are also known as Audit Fields in Salesforce.
Since Winter '16, Salesforce introduces a permission called Set Audit Fields upon Record Creation. By default, a standard profile will not have this permission enabled, such as for System Administrator profile, so you have to create a Permission Set with that permission, then assign the users with that Permission Set.
But, if you do not see this permission, navigate to Setup | Customize | User Interface, and look for a checkbox Enable "Set Audit Fields upon Record Creation" and "Update Records with Inactive Owners" User Permissions", make sure this checkbox is checked.
If you untick this option, the permissions will be disappeared, even it is selected in Permission Set and has been assigned to users.
Once you assigned with "Set Audit Fields upon Record Creation" permission, you should see audit fields like CreatedById, CreatedDate in API tools, such as: Data Loader, Workbench, etc.
This is before have "Set Audit Fields upon Record Creation" permission.
This is after have "Set Audit Fields upon Record Creation" permission.