w3hello.com logo
Home PHP C# C++ Android Java Javascript Python IOS SQL HTML videos Categories
List Highest Correlation Pairs from a Large Correlation Matrix in Pandas?
You can use DataFrame.values to get an numpy array of the data and then use NumPy functions such as argsort() to get the most correlated pairs. But if you want to do this in pandas, you can unstack and order the DataFrame: import pandas as pd import numpy as np shape = (50, 4460) data = np.random.normal(size=shape) data[:, 1000] += data[:, 2000] df = pd.DataFrame(data) c = df.corr().abs() s = c.unstack() so = s.order(kind="quicksort") print so[-4470:-4460] Here is the output: 2192 1522 0.636198 1522 2192 0.636198 3677 2027 0.641817 2027 3677 0.641817 242 130 0.646760 130 242 0.646760 1171 2733 0.670048 2733 1171 0.670048 1000 2000 0.742340 2000 1000 0.742340 dtype: float64

Categories : Python

Create a Correlation Matrix From a Correlation Vector in R
I don't know if there is an automatic way to do this, but expanding on my comment: myvec <- c(-.55, -.48, .66, .47, -.38, -.46) mempty <- matrix(0, nrow = 4, ncol = 4) mindex <- matrix(1:16, nrow = 4, ncol = 4) mempty[mindex[upper.tri(mindex)]] <- myvec mempty[lower.tri(mempty)] <- t(mempty)[lower.tri(t(mempty))] diag(mempty) <- 1 mempty # [,1] [,2] [,3] [,4] # [1,] 1.00 -0.55 -0.48 0.47 # [2,] -0.55 1.00 0.66 -0.38 # [3,] -0.48 0.66 1.00 -0.46 # [4,] 0.47 -0.38 -0.46 1.00 Here's a quickly hacked together function. I hope all my mathematics steps are correct! vec2symmat <- function(invec, diag = 1, byrow = TRUE) { Nrow <- ceiling(sqrt(2*length(invec))) if (!sqrt(length(invec)*2 + Nrow) %% 1 == 0) { stop("invec is wrong length to creat

Categories : R

Merge a branch, but dont fill gitk with noise BUT dont lie about where it came from
Have you already pushed the dev branch and thus shared it with others? If not, an interactive rebase will do an excellent job git checkout devbranch git rebase -i master Git rebase will give you the opportunity to clean up your history by edit commits, squash them together, split them or even skip some. The above will end up putting the commits of devbranch in a straight history after master. If you feel that is too much "lying" then you can rebase devbranch on its own base, so you can rewrite history but still keep it as a branch starting at the same place as before git merge-base master devbranch git rebase -i <hash output of previous command>

Categories : GIT

Create GPRS connection
I don't have any GPRS devices, but it could be as simple as Closing the handles and Releasing resources you have opened. See this similar question: Closing GPRS Connections On Windows Mobile

Categories : C#

android application is not able to access GPRS in dual sim phon
Is there any lines of code related to Network connectivity type in your pgm? Following if a simple method to detect connectivity. public boolean isConnectingToInternet(){ ConnectivityManager connectivity = (ConnectivityManager) getSystemService(Context.CONNECTIVITY_SERVICE); if (connectivity != null) { NetworkInfo[] info = connectivity.getAllNetworkInfo(); if (info != null) for (int i = 0; i < info.length; i++) { Log.d("Connectivity", "info:"+info[i]);//Here you can check network info if available if (info[i].getState() == NetworkInfo.State.CONNECTED) { return true; } } } return false; }

Categories : Android

How to write a service to keep track of gprs data usage on android device
You need to do the following : Create a local database that will store the data usage values . Start a service which runs continuously / periodically to calculate /recalculate the data usage . After data usage is calculated by the service ,add the data into your data usage table . To create local database you can refer to this tutorial on sqlite Here is how you can start a service Creating a Service in Android EDIT There is no way to get notified if any fresh data usage is made .You will have to periodically check it using your service that will run continuously or periodically . You can use the following code to calculate the usage : int UID=Process.myUid(); long recived = TrafficStats.getUidRxBytes(UID); long send = TrafficStats.getUidTxBytes(UID); Other functions that you ca

Categories : Android

NA s in Correlation in R
If your data are in data frame then function cor() will calculate correlation between columns of your two data frame. In your case you get all NA because there is only one row in your data frame. You have to transpose your data frames so that this one row becomes one column and then you can calculate correlation coefficient. To transpose you can use function t(). cor(t(df.A),t(df.B))

Categories : R

Correlation of two arrays in C#
If you don't want to use a third party library, you can use the method from this post (posting code here for backup). double[] array1 = { 3, 2, 4, 5, 6 }; double[] array2 = { 9, 7, 12, 15, 17 }; double correl = Correlation(array1, array2); public double Correlation(double array1, double array2) { double[] array_xy = new double[array1.Length]; double[] array_xp2 = new double[array1.Length]; double[] array_yp2 = new double[array1.Length]; for (int i = 0; i &lt; array1.Length; i++) array_xy[i] = array1[i] * array2[i]; for (int i = 0; i &lt; array1.Length; i++) array_xp2[i] = Math.Pow(array1[i], 2.0); for (int i = 0; i &lt; array1.Length; i++) array_yp2[i] = Math.Pow(array2[i], 2.0); double sum_x = 0; double sum_y = 0;

Categories : C#

NA from correlation function
Plot the data and it should be clear. The data set ## y doesn't vary plot(1:10, rep(10,10)) is just a horizontal line. The correlation coefficient undefined for a horizontal line, since the estimate of the standard deviation for y is 0 (this appears on the denominator of the correlation coefficient). While plot(1:10, 1:10) is the line: y = x

Categories : R

Correlation of an image
Functions like corrcoef return values like [1.0025 -0.0358; -0.0358 0.9975] where the coefficient is -0.0358, and the other values relate to the certainty of the coefficient. Here's what you should expect from the covariance matrix cov: http://www.mathworks.com/help/matlab/ref/cov.html Here's what you should expect from the correlation coefficients corrcoef: http://www.mathworks.com/help/matlab/ref/corrcoef.html What I suspect is going on is there's some variable whose value is affected by running. A useful debugging practice (when going through your code with a fine-toothed comb) is having your code output to the screen some of the values as they are being generated. In Matlab, this is as simple as removing a few ; at the end of a few lines. Hope this helps!

Categories : Image

Correlation computation of non 0 elements
One half of what you are looking for is use = "pairwise.complete.obs" in cor: If use has the value "pairwise.complete.obs" then the correlation or covariance between each pair of variables is computed using all complete pairs of observations on those variables. However, it requires to have NA values instead of zeros, so let us transform our data first: data <- data.frame(x = c(1, 0, -1, 0, 1), y = c(-1, 0, 1, -1, 0), z = c(0, 0, 1, -1, -1)) data # x y z # 1 1 -1 0 # 2 0 0 0 # 3 -1 1 1 # 4 0 -1 -1 # 5 1 0 -1 tempData <- data tempData[tempData == 0] <- NA tempData # x y z # 1 1 -1 NA # 2 NA NA NA # 3 -1 1 1 # 4 NA -1 -1 # 5 1 NA -1 Finally: cor(tempData, use = "pairwise.complete.obs") # x y z # x 1 -1 -1

Categories : R

determining how good a correlation is in R
Here are some of the packages that i know ,are powerfull:- EDIT:- explaning a bit better CART(Classification and regression tree)-rpart package(you can construct decision tree on binary as well as non binary data set depend on the result you require,in your case it is non binary.) BNeT(bayesian network):-deal package(It is based on bayes theorem which defined a causal relationship.) Naive Bayes classifier:-e1071 package,for some basic understanding about Navie Bayes Classifier! There are still many for correlation in R.

Categories : R

How to Get Ticket In Gatling using Correlation
First of all, you will need to make a first GET request to your service as such: http("getLogin") .get(casUrl) Considering the casUrl val contains the path to your actual service, then, and only then, will you be able to retrieve the information you need, let's say, with a css expression: http("getLogin") .get(casUrl) .check(css("input[name='lt']", "value").saveAs("lt")) Checkers are used to extract data from the body of a request. The saveAs is the important part. It will allow you to record data into gatling's session. You can reuse it this way: http("postLogin") .post(...) ... .param("lt", "${lt}") The brackets are also mandatory : it notices Gatling to try and search in the session the value associated with the key lt. Here is a full example based on your script:

Categories : Testing

Correlation between noise and error
The way you had attempted to compute error_x and error_y probably resulted in underestimates due to cancellation of terms, as you were retaining sign information by summing terms abs(x1-x)./x. If you want the mean relative error, then use error_x=mean(abs((x1-x)./x)); error_y=mean(abs((y1-y)./y)); The calculation of the relative (population) standard deviation is an alternative: rstddev_x=sqrt(mean(((x1-x)./x).^2)); rstddev_y=sqrt(mean(((y1-y)./y).^2)); The population standard deviation is another alternative: stddev_x=sqrt(mean((x1-x).^2)); stddev_y=sqrt(mean((y1-y).^2)); Note that division by x and y may lead to instabilities when these become very small numbers. In that sense it may also be better to compare the deformation parameter to the std dev rather than one of the rela

Categories : Matlab

Correlation Loop Function in R
cor accepts two data.frames: A<-data.frame(A1=c(1,2,3,4,5),B1=c(6,7,8,9,10),C1=c(11,12,13,14,15 )) B<-data.frame(A2=c(6,7,7,10,11),B2=c(2,1,3,8,11),C2=c(1,5,16,7,8)) cor(A,B) # A2 B2 C2 # A1 0.9481224 0.9190183 0.459588 # B1 0.9481224 0.9190183 0.459588 # C1 0.9481224 0.9190183 0.459588 diag(cor(A,B)) #[1] 0.9481224 0.9190183 0.4595880 Edit: Here are some benchmarks: Unit: microseconds expr min lq median uq max neval diag(cor(A, B)) 230.292 238.4225 243.0115 255.0295 352.955 100 mapply(cor, A, B) 267.076 281.5120 286.8030 299.5260 375.087 100 unlist(Map(cor, A, B)) 250.053 259.1045 264.5635 275.9035 1146.140 100 Edit2: And some better benchmarks using set.seed(42) A <- as.data.fram

Categories : R

Include a correlation value in web_global_verification
This can be done by saving the correlation value in a string then getting the string value in a Web_Global_verification() function this will look like the below. lr_save_string(lr_eval_string("Exception Thrown on Username : {Username} / Password: {Password},"ExceptionData"); web_global_verification("Text/IC=<Exception>", "Fail=Found", "Search=ALL", "ID={ExceptionData}", LAST);

Categories : Misc

Calculating a GP correlation matrix outside of a loop
I'm wondering if you could cut your calculation and looping times in half by changing these two lines? (Actually the timing was improved by more than 50% 14.304 secs improved to 6.234 secs ) 1: for(j in 1:nrow(X)){ 2: R[i,j] = exp(temp) To: 1: for(j in i:nrow(X)){ 2: R[i,j] = R[j,i]= exp(temp) Tested: > all.equal(R, R2) [1] TRUE That way you populate the lower triangle without doing any calculations.BTW, what's with the 1.99? This is perhaps a problem more suited to submitting as a C program. The Rcpp package supports this and there are a lot of worked examples on SO. Perhaps a search on: [r] rcpp nested loops

Categories : R

pairwise combinations of coefficient and correlation p.value?
You can use the combn function to generate all combinations of column comparisons and then apply across this matrix using cor.test on the combinations of columns of A (this assumes A is available in your global environment): # All combinations of pairwise comparisons cols <- t( combn(1:4,2) ) apply( cols , 1 , function(x) cor.test( A[,x[1] ] , A[ , x[2] ] )$p.value ) #[1] 0.9893876 0.9844555 0.5461623 0.7987615 0.7414658 0.1061751 The pairwise combinations of columns generated by the combn function is: [,1] [,2] [1,] 1 2 [2,] 1 3 [3,] 1 4 [4,] 2 3 [5,] 2 4 [6,] 3 4 Your apply(Y=A,2,allCorr.p,X=A) did not work as expected because (disregarding that you do not need to use Y=A) you pass the whole matrix as the second argument to your function,

Categories : R

WCF : What is the need of Correlation State in Message Inspector?
The first web search hit on your title says: After the user code on the service operation processed the request, and the reply is created into a Message object, BeforeSendReply is called on the inspector code, with the message object and the correlation state, which is whatever AfterReceiveRequest returned – this way the code can correlate the two parts of the message inspection for a single client request.

Categories : Wcf

string "cross correlation" in matlab
With a hat tip to John d'Errico: str1 = 'CGATGATCGATGAATTTTAGCGGATACGATTC'; str2 = 'AACCCGGAAATTTGGAATTTTCCCCAAATACG'; % the circulant matrix n = length(str2); C = str2( mod(bsxfun(@plus,(0:n-1)',0:n-1),n)+1 ); %//' % Find the maximum number of matching characters, and the amount % by which to shift the string to achieve this result [score, shift] = max( sum(bsxfun(@eq, str1, C), 2) ); Faster yes, simpler...well, I'll leave that up to you to decide :) Note that this method trades memory for speed. That is, it creates the matrix of all possible shifts in memory (efficiently), and compares the string to all rows of this matrix. That matrix will contain N² elements, so if N becomes large, it's better to use the loop (or Shai's method).

Categories : Matlab

Records correlation/clustering using Hadoop
For a start, I'm going to assume that you know how to chain MapReduce jobs. If not, see http://developer.yahoo.com/hadoop/tutorial/module4.html#chaining for details. Second, I will assume that you have a distributed key/value store available to you, like Cassandra. Third, your scoring function does not make sense to me. I wouldn't think that "one record from here, one record from there" would let you know that they are the same person. I could believe that "records from here compared to records from there = estimate of whether or not the same person". So I will assume, contrary to your description, that this is how your scoring function actually works. Now what would be a theoretically nice way to solve your problem? Process logs, put into your store a map of unique machine identi

Categories : Algorithm

How to find the correlation coefficients on a matrix of scatterplots in R?
With insight from DWin (1st comment on the question), I went into where the original data was stored (Microsoft Excel) and completely removed the first row. This removed the headers from the data set and allowed R to run computational commands. It also affected the appearance of my scatterplot matrix. Note: Remember that after changing the data in Excel, the file should be saved as a .txt file, text(tab delimited) so that R can read it. The file should then be re-read into R (just enter it again as before).

Categories : R

Shell multiple logs monitoring and correlation
If your logs are going through syslog, and you're using rsyslogd, then you can configure the syslog on each machine to forward the specific messages you're interested in to one (or two) centralized log servers, using a property match like: :msg, contains, "failed password" See the rsyslog documentation for more details about how to set up reliable syslog forwarding.

Categories : Linux

Displaying the numbers in an R correlation matrix diagonally
Doesn't look like cor.plot can pass any arguments through to the text call that plots the numbers. So, you can modify the function but opening the source (edit(cor.plot)) and changing the line text(rx, ry, round(r * 100)) to text(rx, ry, round(r * 100), srt = 45) or however many degrees you want to rotate the numbers by. Then you can copy that code and define a new function (my.cor.plot) with the modified code. I think that there's less hack-ish ways of doing this, but I've never used them and this will work.

Categories : R

How to compute rolling rank correlation using Pandas
I don't think rolling_apply can be used to do a rolling correlation, as it seems to break DataFrames into 1-d arrays. There may be better ways to do this, but one solution is to make generator to yield a slice for each window yourself: def window(length, size=2, start=0): while start + size <= length: yield slice(start, start + size) start += 1 and then loop through it.. In [144]: from pandas import DataFrame ...: import numpy as np ...: ...: df = DataFrame(np.arange(10).reshape(2,5).T, columns=['a','b']) ...: ...: df.iloc[0,1] = -1 #still perfect with ranked correlation, but not with pearson's r ...: ...: for w in window(len(df), size=3): ...: df_win = df.iloc[w,:] ...: spearman = df_win['a'].rank().c

Categories : Python

How to visualize correlation matrix as a schemaball in Matlab
Coincidentally, Cleve Moler (MathWorks Chief Mathematician) showed an example of just this sort of plot on his most recent blog post (not nearly as beautiful as the ones in your example, and the connecting lines are straight rather than parabolic, but it looks functional). Unfortunately he didn't include the code directly, but if you leave him a comment on the post he's usually very willing to share things. What might be even nicer for you is that he also applies (and this time includes) code to permute the rows/columns of the array in order to maximize the spatial proximity of highly connected nodes, rather than randomly ordering them around the circumference. You end up with a 'crescent'-shaped envelope of connecting lines, with the thick bit of the crescent representing the most highly

Categories : Matlab

Rolling correlation between zoo objects of unequal size
Try this: rollapply(inflow, 6, cor, y = outflow) This computes value <- c( cor(inflow[1:6], outflow), cor(inflow[2:7], outflow), ...etc... ) ix <- seq(3, length = length(inflow) - 6 + 1) zoo(value, time(inflow)[ix]) Depending on what you want to get out you may need the align= argument too.

Categories : R

Looping over set of columns to perform spearman correlation analysis
You can use lapply: x <- data[1] lapply(data[-1], cor, x = x, method = "spearman") To do a cor.test, you have to use double brackets to extract the first column as a numeric vector (not as a one-column data frame): x <- data[[1]] lapply(data[-1], cor.test, x = x, method = "spearman")

Categories : R

Does a negative cross correlation show high or low similarity?
-1 is a sign of correlation, too. Only values around 0 are an indication that there is no correlation. Near +1 means, that the image is very similar to the other one. Near -1 means, that it's likely that one image is a negative and should be inverted, so the images are similar and get a correlation near +1.

Categories : Image

Create correlation table for large number of variables
example <- matrix(rep(0.8,25),5,5) Or as @Vincent pointed out, matrix(0.8,5,5) is much better. diag(example) <- 1 > example [,1] [,2] [,3] [,4] [,5] [1,] 1.0 0.8 0.8 0.8 0.8 [2,] 0.8 1.0 0.8 0.8 0.8 [3,] 0.8 0.8 1.0 0.8 0.8 [4,] 0.8 0.8 0.8 1.0 0.8 [5,] 0.8 0.8 0.8 0.8 1.0 Then you can just scale it up to as large as you need. In your case: mat2 <- matrix(0.8,1000,1000)

Categories : R

Print correlation data in same plot position across facets
If you want to place them relative to the corners, you can achieve that by specifying an x or y position of Inf or -Inf: ggplot(d, aes(x=x, y=y)) + geom_text(data=r_df, aes(label=paste("rsq=", rsq)), x=-Inf, y=Inf, hjust=-0.2, vjust=1.2)+ geom_point() + facet_wrap(~l, scales="free") I also adjusted hjust and vjust so the label was not in the exact corner of the graph by pushed away from it a bit.

Categories : R

Pairwise correlation of Pandas DataFrame columns with custom function
You would need to do this in cython for any kind of perf (with a cythonizable function) l = len(df.columns) results = np.zeros((l,l)) for i, ac in enumerate(df): for j, bc in enumerate(df): results[j,i] = func(ac,bc) results = DataFrame(results,index=df.columns,columns=df.columns)

Categories : Python

Adding line of identity to correlation plots using pairs() command in R
I think you just mean something like this: my_line <- function(x,y,...){ points(x,y,...) abline(a = 0,b = 1,...) } pairs(USJudgeRatings, lower.panel = my_line, upper.panel = panel.cor)

Categories : R

Correlation structure corAR1() "not defined" in rpy2 generalised least squares model gls
when calling importr('nlme'), the object returned is like a Python package/namespace. If corAR1() is defined in the nlme package, you should tell Python that it is there: fit = nlme.gls(fmla, cor=nlme.corAR1(value=c(0.5)))

Categories : Python

What is the function that will provide you the lower and upper bounds of correlation coefficient separately?
In the "see also" section of cor's help page, it says cor.test for confidence intervals (and tests) > cor.test(m, h) Pearson's product-moment correlation data: m and h t = 0.8974, df = 4, p-value = 0.4202 alternative hypothesis: true correlation is not equal to 0 95 percent confidence interval: -0.6022868 0.9164582 sample estimates: cor 0.4093729 Or to get at the interval more directly: > x = cor.test(m, h) > x$conf.int [1] -0.6022868 0.9164582 attr(,"conf.level") [1] 0.95

Categories : R

Grouped cross correlation using long data frames, subsetting by row values
The Hmisc library includes the rcorr function that will take a matrix argument require(Hmisc) lapply(split(df[, 3:5], df$B), function(mo) rcorr(as.matrix(mo))[[1]]^2 ) $February site1 site2 site3 site1 1 1 1 site2 1 1 1 site3 1 1 1 $January site1 site2 site3 site1 1 1 1 site2 1 1 1 site3 1 1 1

Categories : R

Fast way to see common observation counts for Python Pandas correlation matrix entries
You can do this, but would need to cythonize (otherwise much slower); however memory footprint should be better (this gives the number of nan observations, your gives number of valid observations, but easily convertible) l = len(df.columns) results = np.zeros((l,l)) mask = pd.isnull(df) for i, ac in enumerate(df): for j, bc in enumerate(df): results[j,i] = (mask[i] & mask[j]).sum() results = DataFrame(results,index=df.columns,columns=df.columns)

Categories : Python

Create covariance matrix from correlation table with both positively and negatively correlated values
set.seed(1) cor.table <- matrix(sample(c(0.1,-0.1),50^2,replace=TRUE),50,50) > isSymmetric(cor.table) [1] FALSE ind <- lower.tri(cor.table) cor.table[ind] <- t(cor.table)[ind] diag(cor.table) <- 1 > isSymmetric(cor.table) [1] TRUE Your problem was you did not create a symmetric matrix. It should work now.

Categories : R

Database design or architecture suitable for storing logs, real time reporting and utilized as log correlation engine
Lots of questions! Q1: Does NoSQL have aggregation? A1: I know Mongo has aggregation, but the last time I used it, it wasn't particularly fast compared to relational databases. Can't speak to Cassandra. Lots of people use Mongo to store structured logs and report. Q2: What about data warehouses? A2: You're right that a data warehouse can exist in a relational database. It's just a different way of structuring the data and thinking about it. Have you thought about keeping a snapshot of time in a real time relational database and then archiving older logs? For example, maybe at 10 million, you start shipping out the oldest log entries to a data warehouse and this guarantees that you are always only looking at the most recent 10 million log entries, which should be fast.

Categories : Database

Why I dont get what I'm looking for
When you return it ends the function. If you want all the users, you'll want to return that array outside of your foreach loop. But since $users is already an array, you don't even need the foreach loop, aside from doing the explode.

Categories : PHP



© Copyright 2017 w3hello.com Publishing Limited. All rights reserved.