INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

Inserting large amount of records

Inserting large amount of records

(OP)
Hi,

We have a process that loads a Microsoft Access database. The software is written in VB 2010 32-bit and using OLEdB. The software runs on Windowds XP and Windows 7. My concern is the amount of time to load the database. There are seven tables and the amount of records inserted depends on the users request. Currently, I am testing with 50,300 records and it takes 11 minutes to insert these records. Each table varies with the number of columns. I need to improve the performance of this process and do not know where to start. Below is a sample of the code:

Dim strSQL As String

strSQL = "INSERT INTO FI(" & _
"[TGSN], " & _
"[MEAS_PRD], " & _
"[BASE_YEAR], " & _
"[REQ_MO1], " & _
"[REQ_TRKS1], " & _
"[REQ_MO2], " & _
"[REQ_TRKS2], " & _
"[REQ_MO3], " & _
"[REQ_TRKS3], " & _
"[REQ_MO4], " & _
"[REQ_TRKS4], " & _
"[REQ_MO5], " & _
"[REQ_TRKS5], " & _
"[REQ_MO6], " & _
"[REQ_TRKS6]) " & _
"VALUES('" & Downloaded_Rec_FI.tgsn & "', '" & _
Downloaded_Rec_FI.meas_prd & "', '" & _
Downloaded_Rec_FI.Base_Year & "', '" & _
arrForecastYears(0).Req_Mo & "', '" & _
arrForecastYears(0).Req_Trks & "', '" & _
arrForecastYears(1).Req_Mo & "', '" & _
arrForecastYears(1).Req_Trks & "', '" & _
arrForecastYears(2).Req_Mo & "', '" & _
arrForecastYears(2).Req_Trks & "', '" & _
arrForecastYears(3).Req_Mo & "', '" & _
arrForecastYears(3).Req_Trks & "', '" & _
arrForecastYears(4).Req_Mo & "', '" & _
arrForecastYears(4).Req_Trks & "', '" & _
arrForecastYears(5).Req_Mo & "', '" & _
arrForecastYears(5).Req_Trks & "')"

Using cmd As New OleDbCommand(strSQL, cnnNew)

cmd.ExecuteNonQuery()
cmd.Dispose()

End Using

------------------------------- END OF CODE ------------------------------------------
All Inserts into table are very similar in nature.


Any ideas?

Thanks!!

The process is written in

RE: Inserting large amount of records


Looks as though you're inserting row by row, which will slow things up considerably. You may want to look SqlBulkCopy.


Mark

"You guys pair up in groups of three, then line up in a circle."
- Bill Peterson, a Florida State football coach

RE: Inserting large amount of records

(OP)
Mark,

Thanks for your reply. I did some research and it is possible I can use the SqlBulkCpy but im my case with Column Headings. I do have a question, can I use this command to load the date to a Microsoft Access data base?

Thanks again,

Fred

RE: Inserting large amount of records

You should be able to since you use OLEDb, but I admit, I've never tried with Access though.

I usually create a datatable to match the destination table which makes a lot of the upfront work a little easier. Here is an example that might get you started.

CODE

Imports System.Data.SqlClient
.
.
.


' set connection to Access 
Dim sqlConn As New SqlConnection
sqlConn.ConnectionString("[Your Connection String]")

Dim sqlBulk As SqlBulkCopy = New SqlBulkCopy(sqlConn)

' Create an import datatable
Dim importDataTable As New DataTable

' can use the same column names
importDataTable.Columns.Add("TGSN", GetType(String))
importDataTable.Columns.Add("MEAS_PRD", GetType(String))
importDataTable.Columns.Add("BASE_YEAR", GetType(String))
importDataTable.Columns.Add("REQ_MO1", GetType(String))
importDataTable.Columns.Add("REQ_TRKS1", GetType(String))
importDataTable.Columns.Add("REQ_MO2", GetType(String))
importDataTable.Columns.Add("REQ_TRKS2", GetType(String))
importDataTable.Columns.Add("REQ_MO3", GetType(String))
importDataTable.Columns.Add("REQ_TRKS3", GetType(String))
importDataTable.Columns.Add("REQ_MO1", GetType(String))
importDataTable.Columns.Add("REQ_TRKS3", GetType(String))
importDataTable.Columns.Add("REQ_MO4", GetType(String))
importDataTable.Columns.Add("REQ_TRKS4", GetType(String))
importDataTable.Columns.Add("REQ_MO5", GetType(String))
importDataTable.Columns.Add("REQ_TRKS5", GetType(String))


' Populate your import datatable with your array/records data
' as you would want to see it in the destination table
For Each row In YourArray ' or however your source data is configured
         ' add a row to the import datatable
         rw = importDataTable.Rows.Add()

	rw("TGSN") = Downloaded_Rec_FI.tgsn
	rw("MEAS_PRD") = Downloaded_Rec_FI.tgsn
	rw("BASE_YEAR") = Downloaded_Rec_FI.tgsn
	rw("REQ_MO1") = arrForecastYears(0).Req_Mo
	rw("REQ_TRKS1") = arrForecastYears(0).Req_Trks
	.
	.
	.
Next

'Set the Destination table name
sqlBulk.DestinationTableName = "FI"

' If the source column names are different than the destination 
' you can map the imported data column names to the destination able column names
' any column not mapped either manually this way or exists in both the source and destination 
' tables will be NULL in the destination table
sqlBulk.ColumnMappings.Add("TGSN", "TGSN")
sqlBulk.ColumnMappings.Add("MEAS_PRD", "MEAS_PRD")
sqlBulk.ColumnMappings.Add("BASE_YEAR", "BASE_YEAR")
.
.
.

'write to the table
sqlBulk.WriteToServer(importDataTable) 


Mark

"You guys pair up in groups of three, then line up in a circle."
- Bill Peterson, a Florida State football coach

RE: Inserting large amount of records

From an Access database, you will load your records a lot quicker if you code it in ADO

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close