There is a new API for streaming data directly into BigQuery which may be a
better match for your use case.
Instead of using a job to load data into BigQuery, you can choose to
stream your data into BigQuery one record at a time by using the
tabledata().insertAll() method. This approach enables querying data
without the delay of running a load job. There are several important
trade-offs to consider before choosing an approach.
If the structure of your data changes, you could have BigQuery run over
its tables and update accordingly. Streaming the raw data will give you
most flexibility but at the higher cost of having to post-process the data