List Highest Correlation Pairs from a Large Correlation Matrix in Pandas? 
You can use DataFrame.values to get an numpy array of the data and then use
NumPy functions such as argsort() to get the most correlated pairs.
But if you want to do this in pandas, you can unstack and order the
DataFrame:
import pandas as pd
import numpy as np
shape = (50, 4460)
data = np.random.normal(size=shape)
data[:, 1000] += data[:, 2000]
df = pd.DataFrame(data)
c = df.corr().abs()
s = c.unstack()
so = s.order(kind="quicksort")
print so[4470:4460]
Here is the output:
2192 1522 0.636198
1522 2192 0.636198
3677 2027 0.641817
2027 3677 0.641817
242 130 0.646760
130 242 0.646760
1171 2733 0.670048
2733 1171 0.670048
1000 2000 0.742340
2000 1000 0.742340
dtype: float64

Create a Correlation Matrix From a Correlation Vector in R 
I don't know if there is an automatic way to do this, but expanding on my
comment:
myvec < c(.55, .48, .66, .47, .38, .46)
mempty < matrix(0, nrow = 4, ncol = 4)
mindex < matrix(1:16, nrow = 4, ncol = 4)
mempty[mindex[upper.tri(mindex)]] < myvec
mempty[lower.tri(mempty)] < t(mempty)[lower.tri(t(mempty))]
diag(mempty) < 1
mempty
# [,1] [,2] [,3] [,4]
# [1,] 1.00 0.55 0.48 0.47
# [2,] 0.55 1.00 0.66 0.38
# [3,] 0.48 0.66 1.00 0.46
# [4,] 0.47 0.38 0.46 1.00
Here's a quickly hacked together function. I hope all my mathematics steps
are correct!
vec2symmat < function(invec, diag = 1, byrow = TRUE) {
Nrow < ceiling(sqrt(2*length(invec)))
if (!sqrt(length(invec)*2 + Nrow) %% 1 == 0) {
stop("invec is wrong length to creat

Merge a branch, but dont fill gitk with noise BUT dont lie about where it came from 
Have you already pushed the dev branch and thus shared it with others?
If not, an interactive rebase will do an excellent job
git checkout devbranch
git rebase i master
Git rebase will give you the opportunity to clean up your history by edit
commits, squash them together, split them or even skip some.
The above will end up putting the commits of devbranch in a straight
history after master. If you feel that is too much "lying" then you can
rebase devbranch on its own base, so you can rewrite history but still keep
it as a branch starting at the same place as before
git mergebase master devbranch
git rebase i <hash output of previous command>

Create GPRS connection 
I don't have any GPRS devices, but it could be as simple as Closing the
handles and Releasing resources you have opened.
See this similar question: Closing GPRS Connections On Windows Mobile

android application is not able to access GPRS in dual sim phon 
Is there any lines of code related to Network connectivity type in your
pgm?
Following if a simple method to detect connectivity.
public boolean isConnectingToInternet(){
ConnectivityManager connectivity = (ConnectivityManager)
getSystemService(Context.CONNECTIVITY_SERVICE);
if (connectivity != null)
{
NetworkInfo[] info = connectivity.getAllNetworkInfo();
if (info != null)
for (int i = 0; i < info.length; i++) {
Log.d("Connectivity", "info:"+info[i]);//Here you can
check network info if available
if (info[i].getState() == NetworkInfo.State.CONNECTED)
{
return true;
}
}
}
return false;
}

How to write a service to keep track of gprs data usage on android device 
You need to do the following :
Create a local database that will store the data usage values .
Start a service which runs continuously / periodically to calculate
/recalculate the data usage .
After data usage is calculated by the service ,add the data into your data
usage table .
To create local database you can refer to this tutorial on sqlite
Here is how you can start a service Creating a Service in Android
EDIT
There is no way to get notified if any fresh data usage is made .You will
have to periodically check it using your service that will run continuously
or periodically .
You can use the following code to calculate the usage :
int UID=Process.myUid();
long recived = TrafficStats.getUidRxBytes(UID);
long send = TrafficStats.getUidTxBytes(UID);
Other functions that you ca

NA s in Correlation in R 
If your data are in data frame then function cor() will calculate
correlation between columns of your two data frame. In your case you get
all NA because there is only one row in your data frame.
You have to transpose your data frames so that this one row becomes one
column and then you can calculate correlation coefficient. To transpose you
can use function t().
cor(t(df.A),t(df.B))

Correlation of two arrays in C# 
If you don't want to use a third party library, you can use the method from
this post (posting code here for backup).
double[] array1 = { 3, 2, 4, 5, 6 };
double[] array2 = { 9, 7, 12, 15, 17 };
double correl = Correlation(array1, array2);
public double Correlation(double array1, double array2)
{
double[] array_xy = new double[array1.Length];
double[] array_xp2 = new double[array1.Length];
double[] array_yp2 = new double[array1.Length];
for (int i = 0; i < array1.Length; i++)
array_xy[i] = array1[i] * array2[i];
for (int i = 0; i < array1.Length; i++)
array_xp2[i] = Math.Pow(array1[i], 2.0);
for (int i = 0; i < array1.Length; i++)
array_yp2[i] = Math.Pow(array2[i], 2.0);
double sum_x = 0;
double sum_y = 0;

NA from correlation function 
Plot the data and it should be clear. The data set
## y doesn't vary
plot(1:10, rep(10,10))
is just a horizontal line. The correlation coefficient undefined for a
horizontal line, since the estimate of the standard deviation for y is 0
(this appears on the denominator of the correlation coefficient). While
plot(1:10, 1:10)
is the line:
y = x

Correlation of an image 
Functions like corrcoef return values like [1.0025 0.0358; 0.0358 0.9975]
where the coefficient is 0.0358, and the other values relate to the
certainty of the coefficient.
Here's what you should expect from the covariance matrix cov:
http://www.mathworks.com/help/matlab/ref/cov.html
Here's what you should expect from the correlation coefficients corrcoef:
http://www.mathworks.com/help/matlab/ref/corrcoef.html
What I suspect is going on is there's some variable whose value is affected
by running. A useful debugging practice (when going through your code with
a finetoothed comb) is having your code output to the screen some of the
values as they are being generated. In Matlab, this is as simple as
removing a few ; at the end of a few lines.
Hope this helps!

Correlation computation of non 0 elements 
One half of what you are looking for is use = "pairwise.complete.obs" in
cor:
If use has the value "pairwise.complete.obs" then the correlation or
covariance between each pair of variables is computed using all
complete pairs of observations on those variables.
However, it requires to have NA values instead of zeros, so let us
transform our data first:
data < data.frame(x = c(1, 0, 1, 0, 1),
y = c(1, 0, 1, 1, 0),
z = c(0, 0, 1, 1, 1))
data
# x y z
# 1 1 1 0
# 2 0 0 0
# 3 1 1 1
# 4 0 1 1
# 5 1 0 1
tempData < data
tempData[tempData == 0] < NA
tempData
# x y z
# 1 1 1 NA
# 2 NA NA NA
# 3 1 1 1
# 4 NA 1 1
# 5 1 NA 1
Finally:
cor(tempData, use = "pairwise.complete.obs")
# x y z
# x 1 1 1

determining how good a correlation is in R 
Here are some of the packages that i know ,are powerfull:
EDIT: explaning a bit better
CART(Classification and regression tree)rpart package(you can construct
decision tree on binary as well as non binary data set depend on the
result you require,in your case it is non binary.)
BNeT(bayesian network):deal package(It is based on bayes theorem which
defined a causal relationship.)
Naive Bayes classifier:e1071 package,for some basic understanding about
Navie Bayes Classifier!
There are still many for correlation in R.

How to Get Ticket In Gatling using Correlation 
First of all, you will need to make a first GET request to your service as
such:
http("getLogin")
.get(casUrl)
Considering the casUrl val contains the path to your actual service, then,
and only then, will you be able to retrieve the information you need, let's
say, with a css expression:
http("getLogin")
.get(casUrl)
.check(css("input[name='lt']", "value").saveAs("lt"))
Checkers are used to extract data from the body of a request. The saveAs is
the important part. It will allow you to record data into gatling's
session.
You can reuse it this way:
http("postLogin")
.post(...)
...
.param("lt", "${lt}")
The brackets are also mandatory : it notices Gatling to try and search in
the session the value associated with the key lt.
Here is a full example based on your script:

Correlation between noise and error 
The way you had attempted to compute error_x and error_y probably resulted
in underestimates due to cancellation of terms, as you were retaining sign
information by summing terms abs(x1x)./x.
If you want the mean relative error, then use
error_x=mean(abs((x1x)./x));
error_y=mean(abs((y1y)./y));
The calculation of the relative (population) standard deviation is an
alternative:
rstddev_x=sqrt(mean(((x1x)./x).^2));
rstddev_y=sqrt(mean(((y1y)./y).^2));
The population standard deviation is another alternative:
stddev_x=sqrt(mean((x1x).^2));
stddev_y=sqrt(mean((y1y).^2));
Note that division by x and y may lead to instabilities when these become
very small numbers. In that sense it may also be better to compare the
deformation parameter to the std dev rather than one of the rela

Correlation Loop Function in R 
cor accepts two data.frames:
A<data.frame(A1=c(1,2,3,4,5),B1=c(6,7,8,9,10),C1=c(11,12,13,14,15 ))
B<data.frame(A2=c(6,7,7,10,11),B2=c(2,1,3,8,11),C2=c(1,5,16,7,8))
cor(A,B)
# A2 B2 C2
# A1 0.9481224 0.9190183 0.459588
# B1 0.9481224 0.9190183 0.459588
# C1 0.9481224 0.9190183 0.459588
diag(cor(A,B))
#[1] 0.9481224 0.9190183 0.4595880
Edit:
Here are some benchmarks:
Unit: microseconds
expr min lq median uq max neval
diag(cor(A, B)) 230.292 238.4225 243.0115 255.0295 352.955 100
mapply(cor, A, B) 267.076 281.5120 286.8030 299.5260 375.087 100
unlist(Map(cor, A, B)) 250.053 259.1045 264.5635 275.9035 1146.140 100
Edit2:
And some better benchmarks using
set.seed(42)
A < as.data.fram

Include a correlation value in web_global_verification 
This can be done by saving the correlation value in a string then getting
the string value in a Web_Global_verification() function this will look
like the below.
lr_save_string(lr_eval_string("Exception Thrown on Username : {Username} /
Password: {Password},"ExceptionData");
web_global_verification("Text/IC=<Exception>",
"Fail=Found",
"Search=ALL",
"ID={ExceptionData}",
LAST);

Calculating a GP correlation matrix outside of a loop 
I'm wondering if you could cut your calculation and looping times in half
by changing these two lines? (Actually the timing was improved by more than
50% 14.304 secs improved to 6.234 secs )
1: for(j in 1:nrow(X)){
2: R[i,j] = exp(temp)
To:
1: for(j in i:nrow(X)){
2: R[i,j] = R[j,i]= exp(temp)
Tested:
> all.equal(R, R2)
[1] TRUE
That way you populate the lower triangle without doing any
calculations.BTW, what's with the 1.99? This is perhaps a problem more
suited to submitting as a C program. The Rcpp package supports this and
there are a lot of worked examples on SO. Perhaps a search on: [r] rcpp
nested loops

pairwise combinations of coefficient and correlation p.value? 
You can use the combn function to generate all combinations of column
comparisons and then apply across this matrix using cor.test on the
combinations of columns of A (this assumes A is available in your global
environment):
# All combinations of pairwise comparisons
cols < t( combn(1:4,2) )
apply( cols , 1 , function(x) cor.test( A[,x[1] ] , A[ , x[2] ] )$p.value )
#[1] 0.9893876 0.9844555 0.5461623 0.7987615 0.7414658 0.1061751
The pairwise combinations of columns generated by the combn function is:
[,1] [,2]
[1,] 1 2
[2,] 1 3
[3,] 1 4
[4,] 2 3
[5,] 2 4
[6,] 3 4
Your apply(Y=A,2,allCorr.p,X=A) did not work as expected because
(disregarding that you do not need to use Y=A) you pass the whole matrix as
the second argument to your function,

WCF : What is the need of Correlation State in Message Inspector? 
The first web search hit on your title says:
After the user code on the service operation processed the request, and
the reply is created into a Message object, BeforeSendReply is called on
the inspector code, with the message object and the correlation state,
which is whatever AfterReceiveRequest returned – this way the code can
correlate the two parts of the message inspection for a single client
request.

string "cross correlation" in matlab 
With a hat tip to John d'Errico:
str1 = 'CGATGATCGATGAATTTTAGCGGATACGATTC';
str2 = 'AACCCGGAAATTTGGAATTTTCCCCAAATACG';
% the circulant matrix
n = length(str2);
C = str2( mod(bsxfun(@plus,(0:n1)',0:n1),n)+1 ); %//'
% Find the maximum number of matching characters, and the amount
% by which to shift the string to achieve this result
[score, shift] = max( sum(bsxfun(@eq, str1, C), 2) );
Faster yes, simpler...well, I'll leave that up to you to decide :)
Note that this method trades memory for speed. That is, it creates the
matrix of all possible shifts in memory (efficiently), and compares the
string to all rows of this matrix. That matrix will contain N² elements,
so if N becomes large, it's better to use the loop (or Shai's method).

Records correlation/clustering using Hadoop 
For a start, I'm going to assume that you know how to chain MapReduce jobs.
If not, see
http://developer.yahoo.com/hadoop/tutorial/module4.html#chaining for
details.
Second, I will assume that you have a distributed key/value store available
to you, like Cassandra.
Third, your scoring function does not make sense to me. I wouldn't think
that "one record from here, one record from there" would let you know that
they are the same person. I could believe that "records from here compared
to records from there = estimate of whether or not the same person". So I
will assume, contrary to your description, that this is how your scoring
function actually works.
Now what would be a theoretically nice way to solve your problem?
Process logs, put into your store a map of unique machine identi

How to find the correlation coefficients on a matrix of scatterplots in R? 
With insight from DWin (1st comment on the question), I went into where the
original data was stored (Microsoft Excel) and completely removed the first
row. This removed the headers from the data set and allowed R to run
computational commands. It also affected the appearance of my scatterplot
matrix.
Note: Remember that after changing the data in Excel, the file should be
saved as a .txt file, text(tab delimited) so that R can read it. The file
should then be reread into R (just enter it again as before).

Shell multiple logs monitoring and correlation 
If your logs are going through syslog, and you're using rsyslogd, then you
can configure the syslog on each machine to forward the specific messages
you're interested in to one (or two) centralized log servers, using a
property match like:
:msg, contains, "failed password"
See the rsyslog documentation for more details about how to set up reliable
syslog forwarding.

Displaying the numbers in an R correlation matrix diagonally 
Doesn't look like cor.plot can pass any arguments through to the text call
that plots the numbers. So, you can modify the function but opening the
source (edit(cor.plot)) and changing the line
text(rx, ry, round(r * 100))
to
text(rx, ry, round(r * 100), srt = 45)
or however many degrees you want to rotate the numbers by. Then you can
copy that code and define a new function (my.cor.plot) with the modified
code.
I think that there's less hackish ways of doing this, but I've never used
them and this will work.

How to compute rolling rank correlation using Pandas 
I don't think rolling_apply can be used to do a rolling correlation, as it
seems to break DataFrames into 1d arrays. There may be better ways to do
this, but one solution is to make generator to yield a slice for each
window yourself:
def window(length, size=2, start=0):
while start + size <= length:
yield slice(start, start + size)
start += 1
and then loop through it..
In [144]: from pandas import DataFrame
...: import numpy as np
...:
...: df = DataFrame(np.arange(10).reshape(2,5).T, columns=['a','b'])
...:
...: df.iloc[0,1] = 1 #still perfect with ranked correlation,
but not with pearson's r
...:
...: for w in window(len(df), size=3):
...: df_win = df.iloc[w,:]
...: spearman = df_win['a'].rank().c

How to visualize correlation matrix as a schemaball in Matlab 
Coincidentally, Cleve Moler (MathWorks Chief Mathematician) showed an
example of just this sort of plot on his most recent blog post (not nearly
as beautiful as the ones in your example, and the connecting lines are
straight rather than parabolic, but it looks functional). Unfortunately he
didn't include the code directly, but if you leave him a comment on the
post he's usually very willing to share things.
What might be even nicer for you is that he also applies (and this time
includes) code to permute the rows/columns of the array in order to
maximize the spatial proximity of highly connected nodes, rather than
randomly ordering them around the circumference. You end up with a
'crescent'shaped envelope of connecting lines, with the thick bit of the
crescent representing the most highly

Rolling correlation between zoo objects of unequal size 
Try this:
rollapply(inflow, 6, cor, y = outflow)
This computes
value < c( cor(inflow[1:6], outflow), cor(inflow[2:7], outflow),
...etc... )
ix < seq(3, length = length(inflow)  6 + 1)
zoo(value, time(inflow)[ix])
Depending on what you want to get out you may need the align= argument too.

Looping over set of columns to perform spearman correlation analysis 
You can use lapply:
x < data[1]
lapply(data[1], cor, x = x, method = "spearman")
To do a cor.test, you have to use double brackets to extract the first
column as a numeric vector (not as a onecolumn data frame):
x < data[[1]]
lapply(data[1], cor.test, x = x, method = "spearman")

Does a negative cross correlation show high or low similarity? 
1 is a sign of correlation, too. Only values around 0 are an indication
that there is no correlation. Near +1 means, that the image is very similar
to the other one. Near 1 means, that it's likely that one image is a
negative and should be inverted, so the images are similar and get a
correlation near +1.

Create correlation table for large number of variables 
example < matrix(rep(0.8,25),5,5)
Or as @Vincent pointed out, matrix(0.8,5,5) is much better.
diag(example) < 1
> example
[,1] [,2] [,3] [,4] [,5]
[1,] 1.0 0.8 0.8 0.8 0.8
[2,] 0.8 1.0 0.8 0.8 0.8
[3,] 0.8 0.8 1.0 0.8 0.8
[4,] 0.8 0.8 0.8 1.0 0.8
[5,] 0.8 0.8 0.8 0.8 1.0
Then you can just scale it up to as large as you need. In your case:
mat2 < matrix(0.8,1000,1000)

Print correlation data in same plot position across facets 
If you want to place them relative to the corners, you can achieve that by
specifying an x or y position of Inf or Inf:
ggplot(d, aes(x=x, y=y)) +
geom_text(data=r_df, aes(label=paste("rsq=", rsq)),
x=Inf, y=Inf, hjust=0.2, vjust=1.2)+
geom_point() +
facet_wrap(~l, scales="free")
I also adjusted hjust and vjust so the label was not in the exact corner of
the graph by pushed away from it a bit.

Pairwise correlation of Pandas DataFrame columns with custom function 
You would need to do this in cython for any kind of perf (with a
cythonizable function)
l = len(df.columns)
results = np.zeros((l,l))
for i, ac in enumerate(df):
for j, bc in enumerate(df):
results[j,i] = func(ac,bc)
results = DataFrame(results,index=df.columns,columns=df.columns)

Adding line of identity to correlation plots using pairs() command in R 
I think you just mean something like this:
my_line < function(x,y,...){
points(x,y,...)
abline(a = 0,b = 1,...)
}
pairs(USJudgeRatings, lower.panel = my_line, upper.panel = panel.cor)

Correlation structure corAR1() "not defined" in rpy2 generalised least squares model gls 
when calling importr('nlme'), the object returned is like a Python
package/namespace.
If corAR1() is defined in the nlme package, you should tell Python that it
is there:
fit = nlme.gls(fmla, cor=nlme.corAR1(value=c(0.5)))

What is the function that will provide you the lower and upper bounds of correlation coefficient separately? 
In the "see also" section of cor's help page, it says
cor.test for confidence intervals (and tests)
> cor.test(m, h)
Pearson's productmoment correlation
data: m and h
t = 0.8974, df = 4, pvalue = 0.4202
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.6022868 0.9164582
sample estimates:
cor
0.4093729
Or to get at the interval more directly:
> x = cor.test(m, h)
> x$conf.int
[1] 0.6022868 0.9164582
attr(,"conf.level")
[1] 0.95

Grouped cross correlation using long data frames, subsetting by row values 
The Hmisc library includes the rcorr function that will take a matrix
argument
require(Hmisc)
lapply(split(df[, 3:5], df$B),
function(mo) rcorr(as.matrix(mo))[[1]]^2 )
$February
site1 site2 site3
site1 1 1 1
site2 1 1 1
site3 1 1 1
$January
site1 site2 site3
site1 1 1 1
site2 1 1 1
site3 1 1 1

Fast way to see common observation counts for Python Pandas correlation matrix entries 
You can do this, but would need to cythonize (otherwise much slower);
however
memory footprint should be better (this gives the number of nan
observations, your gives number of valid observations, but easily
convertible)
l = len(df.columns)
results = np.zeros((l,l))
mask = pd.isnull(df)
for i, ac in enumerate(df):
for j, bc in enumerate(df):
results[j,i] = (mask[i] & mask[j]).sum()
results = DataFrame(results,index=df.columns,columns=df.columns)

Create covariance matrix from correlation table with both positively and negatively correlated values 
set.seed(1)
cor.table < matrix(sample(c(0.1,0.1),50^2,replace=TRUE),50,50)
> isSymmetric(cor.table)
[1] FALSE
ind < lower.tri(cor.table)
cor.table[ind] < t(cor.table)[ind]
diag(cor.table) < 1
> isSymmetric(cor.table)
[1] TRUE
Your problem was you did not create a symmetric matrix. It should work now.

Database design or architecture suitable for storing logs, real time reporting and utilized as log correlation engine 
Lots of questions!
Q1: Does NoSQL have aggregation?
A1: I know Mongo has aggregation, but the last time I used it, it wasn't
particularly fast compared to relational databases. Can't speak to
Cassandra. Lots of people use Mongo to store structured logs and report.
Q2: What about data warehouses?
A2: You're right that a data warehouse can exist in a relational database.
It's just a different way of structuring the data and thinking about it.
Have you thought about keeping a snapshot of time in a real time relational
database and then archiving older logs?
For example, maybe at 10 million, you start shipping out the oldest log
entries to a data warehouse and this guarantees that you are always only
looking at the most recent 10 million log entries, which should be fast.

Why I dont get what I'm looking for 
When you return it ends the function. If you want all the users, you'll
want to return that array outside of your foreach loop. But since $users is
already an array, you don't even need the foreach loop, aside from doing
the explode.


